Microhubs and its applications

ABSTRACT

A system and method of crawling at least one website comprising at least one URL includes maintaining a lookup structure comprising all of the URLs known to be on a website; calculating a hub score for each webpage of the website to be recrawled, wherein the hub score measures how likely the to be recrawled webpage includes links to fresh content published on the website; sorting all the to be recrawled pages by their hub scores; and crawling the to be recrawled pages in order from highest hub scores to lowest hub scores. The calculating comprises computing a first value equaling a percentage of a number of new relative URLs on the to be recrawled page; computing a second value equaling a percentage of a previous hub score of the to be recrawled page; and computing the hub score as a sum of the first and the second values.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. application Ser. No.11/241,469, U.S. Pat. No. 7,496,557, filed Sep. 30, 2005, the completedisclosure of which, in its entirety, is herein incorporated byreference.

BACKGROUND

1. Field of the Invention

The embodiments of the invention generally relate to web searching, and,more particularly, to microhubs used for web searching.

2. Description of the Related Art

Often in web crawling, many irrelevant and unimportant websites arediscovered during the course of searching websites. Therefore, thereremains a need for a novel web crawling technique which reduces theamount of irrelevant websites which are discovered during web crawling.

SUMMARY

In view of the foregoing, the embodiments of the invention provide amethod of crawling at least one website comprising at least oneuniversal resource locator (URL), and a program storage device readableby computer, tangibly embodying a program of instructions executable bythe computer to perform the method of crawling at least one websitecomprising at least one URL, wherein the method comprises maintaining alookup structure comprising all of the URLs known to be on a website;calculating a hub score for each webpage of the website to be recrawled,wherein the hub score measures how likely the to be recrawled webpageincludes links to fresh content published on the website; sorting allthe to be recrawled pages by their hub scores; and crawling the to berecrawled pages in an order from highest hub scores to lowest hubscores.

Preferably, the calculating process comprises computing a first value,wherein the first value equals a percentage of a number of new relativeURLs on the to be recrawled page; computing a second value, wherein thesecond value equals a percentage of a previous hub score of the to berecrawled page; and computing the hub score as a sum of the first valueand the second value. The method may further comprise updating the hubscore by combining a history of the hub scores in a current hub scorecalculation. Moreover, the crawling process is preferably performed by acrawler, and wherein the crawler is preferably adapted to maintain thelookup structure including a history of all URLs familiar to thecrawler; compare crawled URLs with the history; and compute the hubscore based on newly discovered URLs unlisted in the history.

Additionally, when the crawler selects the newly discovered URLs tocrawl, the crawler is preferably adapted to sort eligible URLs by a dateof creation within an immediate previous 24 hour period. Furthermore,the method may comprise recording heuristic information when an URL iscreated, wherein the crawler assigns a creation date and records a lastcrawled date of a discoverer of the newly discovered URL.

Another embodiment of the invention provides a system of crawling atleast one website comprising at least one URL, wherein the systemcomprises a lookup structure comprising all of the URLs known to be on awebsite; a calculator adapted to calculate a hub score for each webpageof the website to be recrawled, wherein the hub score measures howlikely the to be recrawled webpage includes links to fresh contentpublished on the website; a sorter adapted to sort all the to berecrawled pages by their hub scores; and a crawler adapted to crawl theto be recrawled pages in an order from highest hub scores to lowest hubscores. Preferably, the calculator comprises a first processor adaptedto compute a first value, wherein the first value equals a percentage ofa number of new relative URLs on the to be recrawled page; a secondprocessor adapted to compute a second value, wherein the second valueequals a percentage of a previous hub score of the to be recrawled page;and a third processor adapted to compute the hub score as a sum of thefirst value and the second value.

The system may further comprise a mechanism adapted to update the hubscore by combining a history of the hub scores in a current hub scorecalculation. Additionally, the crawler is preferably adapted to maintainthe lookup structure including a history of all URLs familiar to thecrawler; compare crawled URLs with the history; and compute the hubscore based on newly discovered URLs unlisted in the history.Preferably, when the crawler is adapted to select the newly discoveredURLs to crawl, the crawler is adapted to sort eligible URLs by a date ofcreation within an immediate previous 24 hour period. Moreover, thesystem may further comprise a storage unit adapted to record heuristicinformation when an URL is created, wherein the crawler assigns acreation date and records a last crawled date of a discoverer of thenewly discovered URL.

These and other aspects of the embodiments of the invention will bebetter appreciated and understood when considered in conjunction withthe following description and the accompanying drawings. It should beunderstood, however, that the following descriptions, while indicatingpreferred embodiments of the invention and numerous specific detailsthereof, are given by way of illustration and not of limitation. Manychanges and modifications may be made within the scope of theembodiments of the invention without departing from the spirit thereof,and the embodiments of the invention include all such modifications.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments of the invention will be better understood from thefollowing detailed description with reference to the drawings, in which:

FIG. 1 is a flow diagram illustrating a preferred method according to anembodiment of the invention;

FIG. 2 illustrates a system diagram according to an embodiment of theinvention; and

FIG. 3 illustrates a computer architecture diagram according to anembodiment of the invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The embodiments of the invention and the various features andadvantageous details thereof are explained more fully with reference tothe non-limiting embodiments that are illustrated in the accompanyingdrawings and detailed in the following description. It should be notedthat the features illustrated in the drawings are not necessarily drawnto scale. Descriptions of well-known components and processingtechniques are omitted so as to not unnecessarily obscure theembodiments of the invention. The examples used herein are intendedmerely to facilitate an understanding of ways in which the embodimentsof the invention may be practiced and to further enable those of skillin the art to practice the embodiments of the invention. Accordingly,the examples should not be construed as limiting the scope of theembodiments of the invention.

As mentioned, there remains a need for a novel web crawling techniquewhich reduces the amount of irrelevant websites which are discoveredduring web crawling. The embodiments of the invention achieve this byproviding a web crawling technique that utilizes hub scores to determinethe order in which web pages are selected. Referring now to thedrawings, and more particularly to FIGS. 1 through 3, there are shownpreferred embodiments of the invention.

FIG. 1 illustrates a flow diagram of a method of crawling at least onewebsite comprising at least one URL, wherein the method comprisesmaintaining (101) a lookup structure comprising all of the URLs known tobe on a website; calculating (103) a hub score for each webpage of thewebsite to be recrawled, wherein the hub score measures how likely theto be recrawled webpage includes links to fresh content published on thewebsite; sorting (105) all the to be recrawled pages by their hubscores; and crawling (107) the to be recrawled pages in an order fromhighest hub scores to lowest hub scores.

Preferably, the calculating process (103) comprises computing a firstvalue, wherein the first value equals a percentage of a number of newrelative URLs on the to be recrawled page; computing a second value,wherein the second value equals a percentage of a previous hub score ofthe to be recrawled page; and computing the hub score as a sum of thefirst value and the second value. The method may further compriseupdating the hub score by combining a history of the hub scores in acurrent hub score calculation. Moreover, the crawling process (107) ispreferably performed by a crawler, and wherein the crawler is preferablyadapted to maintain the lookup structure including a history of all URLsfamiliar to the crawler; compare crawled URLs with the history; andcompute the hub score based on newly discovered URLs unlisted in thehistory.

Additionally, when the crawler selects the newly discovered URLs tocrawl, the crawler is preferably adapted to sort eligible URLs by a dateof creation. For example, retrieve all URLs discovered within the last(previous) 24 hour period. Furthermore, the method may compriserecording heuristic information when an URL is created, wherein thecrawler assigns a creation date and records a last crawled date of adiscoverer of the newly discovered URL.

Microhubs may be thought of as important links in a website, which leadsto the discovery of the remaining universal resource locators (URLs)belonging to a website. Hereinafter, the term “hub” and “microhub” areused interchangeably. Microhubs can be active or passive. Activemicrohubs include links to newly authored content and are rapidlychanging while passive microhubs include relatively old information anddo not change as often as active microhubs.

For example, the first page in a newspaper website can be thought of asan active hub, since it includes links to headline stories (freshcontent). Archive pages containing links to old stories are examples ofpassive hubs. One property of active microhubs is that the links theyhold are transient in nature. Thus, if one were to take a snapshot of awebsite and perform link analysis, it is very likely that archive pagescontain more links than active microhubs.

Active microhubs represent hotspots of change in the web, which hostslink to fresh content published within a website. Accordingly, activemicrohubs may be very useful in many scenarios. Applications such asbuzz or reputation management inherently rely on newly published contentand big websites containing millions of URLs generally cannot be crawledin timely fashion because of various types of constraints (i.e.,politeness). For these kinds of applications active microhubs present anattractive way to prioritize and crawl the freshly created content. Anactive microhub score (also referred to herein as “hub score”) is ameasure of how likely a URL is to contain links to fresh contentpublished in that site. Fresh content is primarily differentiated interms of how long ago (i.e., timewise) the content was authored. Thehigher the hub score, the more likely that URL will contain links tofresh content. This score is local to a site and is not a global scoreacross the websites.

In accordance with the embodiments of the invention, the hub score of awebpage is calculated based on the number of new relative URLsdiscovered when a page is recrawled. In this context, recrawling meanscrawling the contents of the page once again. The hub score computationcomprises two parts. When a page is recrawled the crawler computes anumber which is a certain percentage (for example 30 percent) of thenumber of new relative URLs it discovers off of that link. This numberis added to another number which is a corresponding percentage (forexample 70 percent) of the previous hub score to compute a new hubscore. The values are chosen in such a way to bias pages whichcontinually hosts new content, while allowing for detection of newmicrohubs in a timely manner. These values can be tuned for variouswebsites. Thus, the hub score can be updated without maintaininghistorical information; the history is combined in the current hubscore. Each time the new hub score is calculated, one may take intoaccount the historical value of the hub score by computing a numberwhich is, for example, 70 percent of the previous hub score.

In other words, mathematically, the hub score=0.7×old hub score+0.3×newrelative links. The cost of computing the hub score is minimal since itis easily adaptable within a webcrawler flow. The crawler maintains alookup structure containing all of the URLs it already knows. Since thewebcrawler partitions URLs by site, this lookup structure is guaranteedto include all on-site URLs known to the entire system. Generally, webscale crawlers are run on distributed systems containing many differentcomputers and since the webcrawler partitions URLs by site, a singlecomputer is guaranteed to contain all the URLs it knows from a site.Therefore, it is relatively easy to detect newly discovered URLs andcompute the hub score in this environment.

A webcrawler may be constrained by politeness, and there is a limit tohow much data can be crawled in a certain time period. Politenessdetermines how often one can crawl a website without overwhelming thecrawled website. There are no hard and fast rules as to how often awebcrawler should crawl a particular website. However, the general ruleof thumb is the webcrawler issues crawl requests approximately onceevery 5 seconds. For large websites comprising millions of URLs it ispreferred that the webcrawler prioritize and crawl only important andrelevant URLs. Applications such as buzz and reputation managementinherently rely on new data. For these applications it is preferred thatthe webcrawler fetches newly published content before any otherdocument.

One of the primary challenges of the webcrawler is the ability to beable to identify and crawl those websites comprising newly publishedcontent, without involving costly analysis. The hub score as provided bythe embodiments of the invention is utilized in this context, in that itprovides a mechanism to identify pages, which are more likely to containlinks to fresh content. A webcrawler may include knobs, which can beused to control the time spent in crawling new pages as opposed to oldpages. When selecting old pages to recrawl, the crawler sorts thecandidate old pages by hub score and crawls them. Crawling of these hubpages ensures that the URLs containing newly published content arediscovered by the crawler. When selecting new URLs to crawl, the crawlersorts the eligible URLs by the date of creation and selects URLsdiscovered within an immediate previous 24 hour period. This ensuresthat the crawler crawls the newly published content before crawlingother documents.

When a website adds a new section or if it gets reorganized, it couldresult in the emergence of new active microhubs. Implicit in thediscussion is how to deemphasize the scores of old hubs. Based on theabove formulae, when a new hub emerges, the scores of the old hub arereduced every time the crawler visits and it discovers that there are nonew relative outlinks emerging from it. Because of the above formulae,the hub score of a page is continually reduced to, for example, 70percent (or whatever the corresponding percentage is calculated to be)of its previous value whenever the crawler revisits (recrawls) an URL.

Determining the approximate date during which a web document was createdis a significant piece of information which can be exploited by higherlevel miners. The crawler records two pieces of information when an URLis created. It assigns a creation date and also records the last crawleddate of the discoverer. These two dates are significant heuristics whichcan be used to find the date in which the page was created. It is likelyto satisfy the following inequality: “Discoverer Last crawldate<Datepage was published<creationdate”.

In this context, “likely to satisfy” is chosen because the page may haveappeared earlier, but may have been linked to/from the hub onlyrecently. Due to the nature of active hubs as places where websitespublish links to their new content, this is an unlikely scenario, andusually the inequality provides useful bounds on the publication date ofa newly-discovered page. This inequality is true even if the hub scoreis not used, but crawling by the fresh content crawling tends to producetighter bounds since good hubs are recrawled often, shrinking thedistance between the “creationdate” and the discoverer's last“crawldate”. Tighter bounds are preferred because it is much more usefulto say that a page first appeared on the web within the last 10 days, asopposed to saying that a page appeared within the last 6 months.

The webcrawler also records how far away a URL is from a hub. This,along with hub score and date of page information, can be used as aheuristic for ranking websites in intranet search engines. This does notin itself provide absolute ranks, rather it can be used as an importantdifferentator when ranking pages. As an example, if two pages have thesame rank and one of the pages happens to have a recent date of pageinformation, then it is very likely that the user is interested in therecent page. The combination of hostrank along with hub scores can pointto hotspots in the web. Here, hostrank determines how important a website is. It is similar to page rank but is calculated with links thatare collapsed at the site level. By sampling these URLs at a predictableinterval, one can find out how news spreads across the web. This canoccur, for example, once every day.

FIG. 2 illustrates a system diagram according to an embodiment of theinvention, wherein the system 200 of crawling at least one websitecomprising at least one URL comprises a lookup structure 201 comprisingall of the URLs known to be on a website; a calculator 203 adapted tocalculate a hub score for each webpage of the website to be recrawled,wherein the hub score measures how likely the to be recrawled webpageincludes links to fresh content published on the website; a sorter 205adapted to sort all the to be recrawled pages by their hub scores; and acrawler 207 adapted to crawl the to be recrawled pages in an order fromhighest hub scores to lowest hub scores.

Preferably, the calculator 203 comprises a first processor 209 adaptedto compute a first value, wherein the first value equals a percentage ofa number of new relative URLs on the to be recrawled page; a secondprocessor 211 adapted to compute a second value, wherein the secondvalue equals a percentage of a previous hub score of the to be recrawledpage; and a third processor 213 adapted to compute the hub score as asum of the first value and the second value.

The system 200 may further comprise a mechanism 215 adapted to updatethe hub score by combining a history of the hub scores in a current hubscore calculation. Additionally, the crawler 207 is preferably adaptedto maintain the lookup structure including a history of all URLsfamiliar to the crawler 207; compare crawled URLs with the history; andcompute the hub score based on newly discovered URLs unlisted in thehistory. Preferably, when the crawler 207 is adapted to select the newlydiscovered URLs to crawl, the crawler 207 is adapted to sort eligibleURLs by a date of creation within an immediate previous 24 hour period.Moreover, the system 200 may further comprise a storage unit 217 adaptedto record heuristic information when an URL is created, wherein thecrawler 207 assigns a creation date and records a last crawled date of adiscoverer of the newly discovered URL.

The embodiments of the invention can take the form of an entirelyhardware embodiment, an entirely software embodiment or an embodimentincluding both hardware and software elements. In a preferredembodiment, the invention is implemented in software, which includes butis not limited to firmware, resident software, microcode, etc.

Furthermore, the embodiments of the invention can take the form of acomputer program product accessible from a computer-usable orcomputer-readable medium providing program code for use by or inconnection with a computer or any instruction execution system. For thepurposes of this description, a computer-usable or computer readablemedium can be any apparatus that can comprise, store, communicate,propagate, or transport the program for use by or in connection with theinstruction execution system, apparatus, or device.

The medium can be an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system (or apparatus or device) or apropagation medium. Examples of a computer-readable medium include asemiconductor or solid state memory, magnetic tape, a removable computerdiskette, a random access memory (RAM), a read-only memory (ROM), arigid magnetic disk and an optical disk. Current examples of opticaldisks include compact disk-read only memory (CD-ROM), compactdisk-read/write (CD-R/W) and DVD.

A data processing system suitable for storing and/or executing programcode will include at least one processor coupled directly or indirectlyto memory elements through a system bus. The memory elements can includelocal memory employed during actual execution of the program code, bulkstorage, and cache memories which provide temporary storage of at leastsome program code in order to reduce the number of times code must beretrieved from bulk storage during execution.

Input/output (I/O) devices (including but not limited to keyboards,displays, pointing devices, etc.) can be coupled to the system eitherdirectly or through intervening I/O controllers. Network adapters mayalso be coupled to the system to enable the data processing system tobecome coupled to other data processing systems or remote printers orstorage devices through intervening private or public networks. Modems,cable modem and Ethernet cards are just a few of the currently availabletypes of network adapters.

A representative hardware environment for practicing the embodiments ofthe invention is depicted in FIG. 3. This schematic drawing illustratesa hardware configuration of an information handling/computer system inaccordance with the embodiments of the invention. The system comprisesat least one processor or central processing unit (CPU) 10. The CPUs 10are interconnected via system bus 12 to various devices such as a randomaccess memory (RAM) 14, read-only memory (ROM) 16, and an input/output(I/O) adapter 18. The I/O adapter 18 can connect to peripheral devices,such as disk units 11 and tape drives 13, or other program storagedevices that are readable by the system. The system can read theinventive instructions on the program storage devices and follow theseinstructions to execute the methodology of the embodiments of theinvention. The system further includes a user interface adapter 19 thatconnects a keyboard 15, mouse 17, speaker 24, microphone 22, and/orother user interface devices such as a touch screen device (not shown)to the bus 12 to gather user input. Additionally, a communicationadapter 20 connects the bus 12 to a data processing network 25, and adisplay adapter 21 connects the bus 12 to a display device 23 which maybe embodied as an output device such as a monitor, printer, ortransmitter, for example.

The foregoing description of the specific embodiments will so fullyreveal the general nature of the invention that others can, by applyingcurrent knowledge, readily modify and/or adapt for various applicationssuch specific embodiments without departing from the generic concept,and, therefore, such adaptations and modifications should and areintended to be comprehended within the meaning and range of equivalentsof the disclosed embodiments. It is to be understood that thephraseology or terminology employed herein is for the purpose ofdescription and not of limitation. Therefore, while the embodiments ofthe invention have been described in terms of preferred embodiments,those skilled in the art will recognize that the embodiments of theinvention can be practiced with modification within the spirit and scopeof the appended claims.

1. A method of crawling at least one website comprising at least oneuniversal resource locator (URL), said method comprising: maintaining alookup structure comprising all of the URLs known to be on a website;calculating a hub score for each webpage of said website to berecrawled, wherein said hub score measures how likely the to berecrawled webpage includes links to fresh content published on saidwebsite, wherein said fresh content is differentiated from other contentbased on how long ago said fresh content was authored; sorting all saidto be recrawled pages by their hub scores; and crawling said to berecrawled pages in an order from highest hub scores to lowest hub scoresto crawl said fresh content before crawling relatively older publishedcontent.
 2. The method according to claim 1, wherein said crawling isperformed by a crawler, and wherein said crawler is adapted to maintainsaid lookup structure including a history of all URLs familiar to saidcrawler.
 3. The method according the claim 2, wherein said crawler isadapted to compare crawled URLs with said history.
 4. The methodaccording to claim 3, wherein said crawler is adapted to compute saidhub score based on newly discovered URLs unlisted in said history. 5.The method according to claim 4, wherein when said crawler selects saidnewly discovered URLs to crawl, said crawler being adapted to sorteligible URLs by a date of creation within an immediate previous 24 hourperiod.
 6. The method according to claim 4, further comprising recordingheuristic information when an URL is created, wherein said crawlerassigns a creation date and records a last crawled date of a discovererof said newly discovered URL.
 7. A program storage device readable bycomputer, tangibly embodying a program of instructions executable bysaid computer to perform a method of crawling at least one websitecomprising at least one universal resource locator (URL), said methodcomprising: maintaining a lookup structure comprising all of the URLsknown to be on a website; calculating a hub score for each webpage ofsaid website to be recrawled, wherein said hub score measures how likelythe to be recrawled webpage includes links to fresh content published onsaid website, wherein said fresh content is differentiated from othercontent based on how long ago said fresh content was authored; sortingall said to be recrawled pages by their hub scores; and crawling said tobe recrawled pages in an order from highest hub scores to lowest hubscores to crawl said fresh content before crawling relatively olderpublished content.
 8. The program storage device according to claim 7,wherein said crawling is performed by a crawler, and wherein saidcrawler is adapted to maintain said lookup structure including a historyof all URLs familiar to said crawler.
 9. The program storage deviceaccording to claim 8, wherein said crawler is adapted to compare crawledURLs with said history.
 10. The program storage device according toclaim 9, wherein said crawler is adapted to compute said hub score basedon newly discovered URLs unlisted in said history.
 11. The programstorage device according to claim 10, wherein said crawler selects saidnewly discovered URLs to crawl, said crawler being adapted to sorteligible URLs by a date of creation within an immediate previous 24 hourperiod.
 12. The program storage device according to claim 10, furthercomprises recording heuristic information when an URL is created,wherein said crawler assigns a creation date and records a last crawleddate of a discoverer of said newly discovered URL.
 13. A system ofcrawling at least one website comprising at least one universal resourcelocator (URL), said system comprising: a lookup structure comprising allof the URLs known to be on a website; a calculator adapted to calculatea hub score for each webpage of said website to be recrawled, whereinsaid hub score measures how likely the to be recrawled webpage includeslinks to fresh content published on said website, wherein said freshcontent is differentiated from other content based on how long ago saidfresh content was authored; a sorter adapted to sort all said to berecrawled pages by their hub scores; and a crawler adapted to crawl saidto be recrawled pages in an order from highest hub scores to lowest hubscores to crawl said fresh content before crawling relatively olderpublished content.
 14. The system according to claim 13, wherein saidcrawler is adapted to maintain said lookup structure including a historyof all URLs familiar to said crawler.
 15. The system according to claim14, wherein said crawler is adapted to compare crawled URLs with saidhistory.
 16. The system according to claim 15, wherein said crawler isadapted to compute said hub score based on newly discovered URLsunlisted in said history.
 17. The system according to claim 16, whereinwhen said crawler is adapted to select said newly discovered URLs tocrawl, said crawler being adapted to sort eligible URLs by a date ofcreation within an immediate previous 24 hour period.
 18. The systemaccording to claim 16, further comprising a storage unit adapted torecord heuristic information when an URL is created, wherein saidcrawler assigns a creation date and records a last crawled date of adiscoverer of said newly discovered URL.